hysop.tools.io_utils module

Tools related to i/o in HySoP.

• IO

• IOParams

• Writer

• XMF, tools to prepare/write xmf files.

class hysop.tools.io_utils.IO

Bases: object

Static class with utilities to set/find the place where i/o files will be read/written.

ASCII = 997

ascii format id

HDF5 = 998

HDF5 format id

static cache_path()

static check_dir(filepath, io_rank=0, comm=None)

Check if the directory of ‘filename’ exists and creates it if not.

Parameters:

• filepath (string) – directory path with full or relative path

• io_rank (int) – processus rank that does the check.

• comm (mpi communicator) – the mpi communicator that does the check.

classmethod default_cache_path()

classmethod default_path()

Get the current default path used for io.

Returns:

the default value of the current i/o path.

Return type:

string

classmethod default_ram_path()

Get the current default path used for io in memory.

Returns:

the default value of the current RAM i/o path.

Return type:

string

static get_datasetnames(filename)

Return the list of dataset names present in hdf input file

Parameters:

filename (string) – hdf file

Return type:

a list of strings

static get_fs_type(path)

classmethod get_tmp_dir(key)

Create or get an existing temporary directory.

classmethod is_shared_fs(path)

classmethod ram_path()

classmethod set_cache_path(path)

static set_datasetname(field_name, topo, direction=None)

Return the dataset name of a given continuous field, saved for a given topology

static set_default_path(pathdir)

Set a new default path for hysop i/o.

Parameters:

pathdir (string) – the new path

class hysop.tools.io_utils.IOParams(filename, filepath=None, frequency=1, fileformat=None, dump_times=None, dump_tstart=None, dump_tend=None, dump_func=None, io_leader=0, visu_leader=0, with_last=False, enable_ram_fs=False, force_ram_fs=False, dump_is_temporary=False, postprocess_dump=None, hdf5_disable_compression=False, hdf5_disable_slicing=False, append=False, **kwds)

Bases: IOParams

A struct to handle I/O files parameters

Parameters:

• filename (string) – Name of the file (absolute or relative path)

• filepath (string) – Location of the file

• frequency (int) – Frequency of output or input (e.g. every N times steps)

• fileformat (int) – Format of the file. See notes for available format. Default=HDF5.

• dump_times (tuple of floats) – Extra dump times that should be used to dump in addition to frequency (double precision)

• dump_tstart (float) – Start to dump at given time. Defaults to -np.inf (no time constraints).

• dump_tend (float) – Stop to dump at given time. Defaults to +np.inf (no time constraints).

• dump_func (function) – Generic function to compute the should_dump result.

• with_last (boolean) – should dump when iteration is last one

• io_leader (int) – Rank of the mpi process dealing with the io. Default is 0.

• visu_leader (int) – Rank of the mpi process dealing with the graphical io. Default is 0.

• enable_ram_fs (bool) – Instruct the dumper to write directly to RAM, fallback to filepath/filename when this is not possible.

• force_ram_fs (bool) – Force the dumper to write directly to RAM, and raise an error when this is not possible (filepath/filename are ignored). Implies enable_ram_fs.

• dump_is_temporary (bool) – Instruct the dumper to delete dumped data from disk or RAM after postprocessing script has been called. Implies that a postprocessing script is supplied.

• postprocess_dump (str) – Path to a postprocessing script that will be called after dump. See hysop/tools/postprocess_dump.sh for an example of post processing script.

• hdf5_disable_compression (bool) – Disable compression for HDF5 outputs (when available). Can be used to accelerate in RAM postprocessing.

• hdf5_disable_slicing (bool) – Disable slicing for HDF5 outputs (when available). May reduce performance but avoid hdf5 file fragmentation.

• append (bool, optional) – Tell if appended (on xmf files, when using hdf format)

• kwds (dict) – Custom extra keyword arguments to pass to operators

• hysop.operator.hdf_io (See examples in)

Notes

Format parameter must be one of the following :

• HDF5

• ASCII

Create new instance of IOParams(filename, filepath, frequency, fileformat, dump_times_fp32, dump_times_fp64, dump_tstart, dump_tend, dump_func, io_leader, visu_leader, with_last, enable_ram_fs, force_ram_fs, dump_is_temporary, postprocess_dump, append, hdf5_disable_compression, hdf5_disable_slicing, disk_filepath, kwds)

clone(**kwds)

property dump_times

should_dump(simulation)

to_string(prefix='')

class hysop.tools.io_utils.Writer(io_params, buffshape=None, mpi_params=None, safe_io=True)

Bases: object

To write data from a 2D numpy array into an ascii file.

Examples

>>> from hysop.tools.io_utils import IOParams, IO, Writer
>>> params = IOParams(filename='r.dat', fileformat=IO.ASCII)
>>> wr = Writer(params, buffshape=(1, 2))
>>> ite = 3 # current iteration number
>>> if wr.do_write(ite):
...    wr.buffer[...] = 3.
...    wr.write()
>>> wr.finalize()

result : buffer is written into r.dat

Parameters:

• io_params (hysop.tools.io_utils.IOParams) – Setup for file ouput (name, location …)

• buffshape (tuple) – 2D numpy.array.shape like tuple, shape of the output/input buffer.

• mpi_params (hysop.tools.parameters.MPIParams) – Mpi setup (comm that owns the writer)

• safe_io (boolean) – True –> open/close file everytime data are written. False –> open at init and close during finalize. Cost less but if simu crashes, data are lost.

buffer

buffer used to save printed data

do_write(ite)

Returns true if output is required for iteration ite

Parameters:

ite (int) – current iteration number

finalize()

close, if required

class hysop.tools.io_utils.XMF

Bases: object

Static class - Tools to prepare and write xmf file

static prepare_grid_attributes(dataset_names, resolution, origin, step, joinrkfiles=None)

Prepare XDMF header as a string.

Parameters:

• dataset_names (list) – all datasets names

• resolution (3d tuple)

• origin (3d tuple)

• step (3d tuple)

• joinrkfiles ((optional))

• Returns

• --------

• string –

  the xml-like header formattable with the following keywords:

  niteration : iteration number time: time in seconds filename: target file name, in sequential or with parallel hdf5 support filename0, … filenameN : target file names for each rank 0 to N, in parallel without HDF5 parallel support resolution0, … resolutionN : local resolutions for each rank 0 to N, in parallel without HDF5 parallel support